Monolithic Image Perception Device and Method

ABSTRACT

The present invention is directed to an apparatus which can acquire, readout and perceive a scene based on the insertion, or embedding of photosensitive elements into or on a transparent or semi-transparent substrate such as glass or plastic. The substrate itself may act as the optical device which deflects the photons of an incident image into the photosensitive elements. A digital neural memory can be trained to recognize patterns in the incident photons. The photosensitive elements and digital neural memory elements may be arranged with light elements controlled in accordance with the patterns detected. In one application, intelligent lighting units provide light while monitoring surroundings and/or adjusting light according to such surroundings. In another application, intelligent displays display images and/or video while monitoring surroundings and/or adjusting the displayed images and/or video in accordance with such surroundings.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.12/880,964, filed on Sep. 13, 2010, which is a continuation-in-part ofU.S. patent application Ser. No. 11/477,571, filed Jun. 30, 2006, whichclaims the benefit of priority to Provisional Patent Application Ser.No. 60/694,988, filed Jun. 30, 2005, the entire disclosures of which areincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of Invention

The present invention relates generally to imaging devices and methods.In particular, the present invention relates to micro-devices for imagerecognition embedded in or positioned on a transparent orsemi-transparent substrate, such as glass or plastic, with lightelements, and methods for image recognition.

2. Description of the Related Art

Transparent surfaces, such as glass, have existed for hundreds of years.Transparent surfaces were initially aimed at protecting a living spacewhile letting the occupants have the perception of the outside world(landscape, weather and possible threat). More recently, transparentsurfaces are in huge demand for the display industry, beginning withCathode Ray Tubes (CRT) and more recently for Liquid Crystal Displays(LCD) and many other kinds of flat panel displays. In use, in most ofthe cases, a human or living organism (animal, plants) is positionedclose to such transparent surfaces.

Image sensors have been available for a few decades (e.g., CCD or CMOSsensors). For example, see U.S. Pat. No. 6,617,565 for a single chipCMOS image sensor, the contents of which are incorporated herein byreference. Typical image sensors are based on camera designs andgenerally include an integrated circuit located behind a lens, which canbe miniature or removable (e.g., screw mounting lens). Sensors are usedto transform light energy (photons) into an electrical signalproportional to the amount of light received by the photosensitiveelements that are organized into an array on the sensor. An image issynthesized from the output of the photosensitive elements.

Image recognition technology is becoming increasingly in demand. Videocameras of various sizes and makes are in demand for applications suchas security, identification, intelligence, quality inspection, trafficsurveillance and more. Video cameras are very often linked to displaydevices by either a wired or a wireless connection. Today, cell phonesare routinely outfitted with miniature cameras connected to an LCDdisplay device disposed therein.

Advanced image recognition requires high resolution imaging synthesis.Current image recognition systems operate at relatively slow speedsbecause of a lack of processing power and/or because processors can onlyprocess one pixel of an image at a time.

Thus, there is a need for new imaging recognition devices that areimproved over the prior art.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide an image recognitiondevice that has a sensing area (e.g., photosensitive elements) directlyincluded in or on a transparent or semi-transparent materialconstituting the optical interface between the incident image and thesensing area. The image recognition device itself is preferablytransparent or semi-transparent.

It is also another object of the present invention to provide thesensing area with “local” decision capability by means of an array oftrainable processing elements. In one embodiment of the presentinvention, trainable cognitive memory elements or cells are associatedwith one or more photosensitive elements. Local decision capabilityprovides the advantage that it reduces the transmission requirements(i.e., bandwidth) of the device, especially when the number ofphotosensitive elements is large and when the transmission frequency ofthe photosensitive elements must be high. By providing a large array ofsensing areas each having local decision capability, a high-resolution,high-speed imaging device is achievable.

According to an embodiment of the present invention, trainable cognitivememory elements can operate in parallel at low frequency and draw verylow current. As a result, autonomous operation of each element isensured and very economical energy sources, such as a solar cell orequivalent, can be used.

According to an embodiment of the present invention, a novel monolithicimage recognition device is formed by association of one or morephotosensitive elements to one or more trainable cognitive memoryelements, all embedded into or positioned on a substrate.

According to an embodiment of the present invention, a plurality ofphotosensitive elements associated with a plurality of trainablecognitive elements can be arranged in one or multiple arrays and spreadover a flat transparent or semi-transparent substrate. The arrays canhave variable geometry and connectivity. Typical geometry can be, but isnot limited to, a linear array of neurons in parallel, or a2-dimensional array of neurons connected in a raster or honey-combedgeometry.

In one aspect, the present invention provides an image recognitiondevice having a plurality of cognitive sensors, a plurality of opticalinterfaces and a plurality of light elements. The plurality of cognitivesensors is embedded in or positioned on a transparent orsemi-transparent substrate. Each sensor has a photosensitive element anda trainable cognitive memory cell associated with the photosensitiveelement. Each one of the plurality of optical interfaces embedded in orformed on the substrate is optically coupled to a corresponding one ofthe plurality of cognitive sensors. The plurality of light elements arepositioned on the substrate and configured to emit light.

In some embodiments, the light output by one or more of the plurality oflight elements may be controlled in accordance with an output from oneor more of the plurality of cognitive sensors. The plurality of lightelements may be geometrically arranged to form an image displayapparatus. The image display apparatus may be configured to display animage representing light received on the photosensitive elements of oneor more of the plurality of cognitive sensors. Each cognitive sensor maybe trainable and configured to recognize patterns based on incidentlight, and the image display apparatus may be configured to display animage and to modify the image in accordance with patterns recognized byone or more of the plurality of cognitive sensors. The plurality ofcognitive sensors may have a field of view, and the light elements maybe configured to emit light in the field of view. The light elements maybe configured to display an image which is visible by objects in thefield of view. The light elements may be configured to provide lightingto objects in the field of view and to display an image to the objectsin the field of view. One or more of the plurality of cognitive elementsmay be configured to recognize patterns of incident light and to controlthe provision of lighting and image display of one or more of theplurality of light elements as a function of the patterns recognized.Each cognitive memory cell may be taught to recognize a differentportion of an image, and the plurality of cognitive memory cells may beconfigured to operate collectively to recognize the image. Eachcognitive memory element may have a plurality of neurons coupled on aninput side thereof by a multiplexed input bus and on an output sidethereof by an output bus, each neuron being taught with a knowledge thatallows the corresponding neuron to recognize a signal and perform adecision. The plurality of cognitive sensors may be configured toperform image recognition operations digitally without a softwareprogram through a plurality of parallel elements each having selfcontained, autonomous behavior. Light elements of the plurality of lightelements are selected from light emitting diodes (LEDs), organic LEDsand plasma cavities.

In some embodiments, the image recognition device may have photovoltaicdevices embedded in or positioned on the substrate. The photovoltaicdevices may be configured to receive power supplied wirelessly and tosupply the received power to the plurality of cognitive sensors and tothe plurality of light elements. The image recognition device may haveoutput transmission lines and power supply lines that are directlyengraved or diffused on the substrate. Each of the plurality ofcognitive sensors may be configured to receive power from power supplylines and to output communications using the power supply lines. Theimage recognition device may have a transparent or semi-transparentcover layer, and the plurality of cognitive sensors and the plurality oflight elements may be arranged between the cover layer and thesubstrate. One or more of the plurality of light elements may beconfigured to emit light through the substrate. One or more of theplurality of light elements may be configured to emit light through thecover layer. The plurality of light elements and plurality of cognitivesensors may be arranged in rows and columns, and the plurality of lightelements and plurality of cognitive sensors may alternate in each rowand alternate in each column. The plurality of light elements andplurality of cognitive sensors may be arranged so that each of theplurality of cognitive sensors is surrounded by light elements. Theplurality of light elements may include, without limitation, red pixels,green pixels and blue pixels.

In another aspect, the present invention provides an image recognitiondevice having a sensing element, processing element coupled to thesensing element, and light emitting element. The sensing element isembedded in or positioned on a transparent or semi-transparentsubstrate. The processing element is embedded in or positioned on thesubstrate. The light emitting element is embedded in or positioned onthe substrate. The transparent or semi-transparent substrate constitutesan optical interface between an incident image to be sensed and asensing pixel of the sensing element. The light emitting element isconfigured to emit light toward the incident image or away from theincident image. In some embodiments, the light emitting element may beone or more light emitting diodes (LEDs), organic LEDs (OLEDs) or plasmacavities. The lighting element may be controlled selectively by anoutput of the processing element. The processing element may betrainable and configured to recognize patterns based on the sensedincident image. The processing element may be configured to control thelight emitted by the light element in accordance with the patternsrecognized by the processing element. The sensing element may have afield of view, and the light elements may be configured to emit light inthe field of view. The sensing element may have a field of view, and thelight elements may be configured to emit light in a direction away fromthe field of view. The processing element may comprise a plurality ofneurons coupled on an input side thereof by a multiplexed input bus andon an output side thereof by an output bus. Each neuron my be taughtwith a knowledge, and the knowledge may allow the corresponding neuronto recognize a signal and perform a decision. The processing element maybe configured to perform image recognition operations digitally withouta software program through a plurality of parallel elements each havingself contained, autonomous behavior. The image recognition device mayfurther comprise photovoltaic devices embedded in or positioned on saidsubstrate. The image recognition device may further comprise outputtransmission lines and power supply lines that are directly engraved ordiffused on said substrate. The processing element may be configured toreceive power from power supply lines and to output communications usingthe power supply lines. The image recognition device may furthercomprise a transparent or semi-transparent cover layer, and the sensingelement, the processing element and the light element may be arrangedbetween the cover layer and the substrate. The light element may beconfigured to emit light through the substrate. The light element may beconfigured to emit light through the cover layer.

In another aspect, the present invention provides an image recognitionmethod. The method provides an optical path to a plurality of sensingelements embedded in or provided on a transparent or semi-transparentsubstrate by using a plurality of optical interfaces embedded in orprovided on the substrate. The method processes, in parallel, signalsgenerated from the plurality of sensing elements in a plurality ofprocessing elements each coupled to one of the sensing elements and eachembedded in or provided on the substrate. The method emits light from aplurality of light elements embedded in or provided on the substrate.

In some embodiments, the emitting of light may include controlling thelight emitted from the plurality of light elements in accordance withoutputs from one or more of the plurality of processing elements. Theprocessing may include recognizing patterns and the emitting may includecontrolling the light emitted from the plurality of light elements inaccordance with the recognized patterns. The recognizing patterns mayinclude detecting the presence of one or more objects within a field ofview of the plurality of sensing elements. The recognizing patterns mayinclude determining distance from the substrate of the one or moredetected objects. The recognizing patterns may include determining thenumber of the one or more detected objects. The recognizing patterns mayinclude locating the position of the one or more detected objects. Thecontrolling may include emitting a reduced amount of light from aplurality of light elements when the presence of no objects is detected.The recognizing patterns may include determining whether any of the oneor more detected objects is an authorized object. The recognizingpatterns may include locating and tracking the gaze of one or moreviewers within a field of view of the plurality of sensing elements. Therecognizing patterns may include facial recognition or facial expressionrecognition. The recognizing patterns may include biometricidentification. The emitting may include displaying an image. Thedisplayed image may correspond to an image received by the plurality ofsensing elements. The processing may include recognizing patterns, andthe emitting may further include modifying the displayed image inaccordance with the recognized patterns.

In another aspect, the present invention provides an image recognitiondevice having a transparent or semi-transparent substrate, a pluralityof cognitive sensors, a plurality of optical interfaces, a filler layer,and a plurality of light elements. The plurality of cognitive sensors ispositioned on the substrate, and each sensor includes a photosensitiveelement and a trainable cognitive memory cell associated with thephotosensitive element. The plurality of optical interfaces is formed onthe substrate and each are optically coupled to corresponding cognitivesensors. The filler layer has a filler between adjacent cognitivesensors of the plurality of cognitive sensors. The plurality of lightelements is positioned on the filler layer and each is configured toemit light. In some embodiments, the plurality of light elementsincludes red pixels, green pixels and blue pixels.

Further applications and advantages of various embodiments of thepresent invention are discussed below, with reference to the drawingfigures.

In another aspect, the present invention provides an image recognitionmethod. The method provides an optical path to a sensing elementembedded in or provided on a transparent or semi-transparent substrateby using an optical interface embedded in or provided on the substrate.The method processes signals generated from the sensing element in aprocessing element coupled to the sensing element and embedded in orprovided on the substrate. The method emits light from a light elementembedded in or provided on said substrate.

In some embodiments, the emitting may comprise controlling the lightemitted from the light element in accordance with an output from theprocessing element. The processing may comprise recognizing patterns andthe emitting may comprise controlling the light emitted from theplurality of light elements in accordance with the recognized patterns.The recognizing patterns may comprise detecting the presence of one ormore objects within a field of view of said sensing element. Therecognizing patterns may comprise determining distance from saidsubstrate of the one or more detected objects. The recognizing patternsmay comprise determining the number of the one or more detected objects.The recognizing patterns may comprise locating the position of the oneor more detected objects. The recognizing patterns may comprisedetermining whether any of the one or more detected objects is anauthorized object. The recognizing patterns may comprise locating andtracking the gaze of one or more viewers within a field of view of thesensing element. The recognizing patterns may comprise facialrecognition or facial expression recognition. The recognizing patternsmay comprise biometric identification. The emitting may comprisedisplaying an image. The displayed image corresponds to an imagereceived by the sensing element. The processing comprises recognizingpatterns and the emitting further comprises modifying the displayedimage in accordance with the recognized patterns.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B include respectively, a front and top view of an array ofsensors disposed on a glass or plexiglass or other transparent plasticor transparent substrate, having etched lenses therein, according to anembodiment of the present invention;

FIG. 2 is a top view of an array of sensors disposed on a glass orplexus substrate, having etched lenses therein, shown detecting DNAfragments, according to an embodiment of the present invention;

FIGS. 3A-3B illustrate respectively a side and top view of a die ofsensors according to one embodiment of the present invention;

FIG. 4 is a block diagram of a sensors according to an embodiment of thepresent invention;

FIG. 5A is a block diagram of a sensors arrays according to anembodiment of the present invention;

FIG. 5B is a block diagram of a sensors bank of arrays, according to anembodiment of the present invention;

FIGS. 6A-6C illustrate neural configurations according to embodiments ofthe present invention;

FIG. 7 is a block diagram of a neuron according to an embodiment of thepresent invention; and

FIGS. 8-12 illustrate exemplary applications of the image recognitiondevice according to embodiments of the present invention.

FIGS. 13A-13D illustrate embodiments in which sensors and light elementsare disposed in or on the same transparent or semi-transparentsubstrate.

FIGS. 14A-14C illustrate an embodiment of an intelligent lighting unithaving sensors and light elements disposed in or on the same transparentor semi-transparent substrate.

FIGS. 15A and 15B illustrate an embodiment of an intelligent displayhaving sensors and light elements disposed in or on the same transparentor semi-transparent substrate.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

While the present invention may be embodied in many different forms, anumber of illustrative embodiments are described herein with theunderstanding that the present disclosure is to be considered asproviding examples of the principles of the invention and such examplesare not intended to limit the invention to any specific preferredembodiments described and/or illustrated herein.

The present invention is an imaging device that may include a sensorperception device, such as a photosensitive element, connected, bound orotherwise associated with a trainable cognitive element, with bothelements deposited chemically or otherwise on or embedded in the surfaceof a transparent substrate. The association of a sensing area with atrainable cognitive element having “local” decision capability isreferenced throughout this document as a “CogniSensor.” A trainablecognitive element is referenced throughout this document as a“CogniMem.” Sensing areas are generally made up of one or morephotosensitive elements, but other sensing arrangements are cotemplated.

According to embodiments of the present invention, CogniSensors can beconfigured to recognize incoming light patterns (e.g., images orportions of images), process the incoming light patterns to make a localdecision, and transmit results of or an indication of the localdecision. A CogniSensor may include a number of components such as, butnot limited to, local decision capability-data input logic, “neurons”and decision output logic, a memory buffer, solar cells for energyautonomy and more. Each CogniSensor preferably features reactiveassociative learning memories (REALM) arranged in parallel. According toan embodiment of the present invention, CogniMem are capable of patternrecognition without any computer instructions, whether digital oranalog.

CogniMem may comprise one or more neurons, which are associativememories accessible in parallel that can react to input patterns similarto their own contents. Neurons can react individually or collectively byreinforcing their response based on the response of other neighboringneurons. This selection can be made through an Inhibitatory/Excitatoryinput line connected to the neurons.

The contents of the neurons of a CogniMem constitute “knowledge.”Knowledge is a set of statically discriminative digital signatures.Knowledge can be static (loaded once) or dynamic (updated by thereaction of other neurons or loaded adaptively from an externalknowledge base), but is preferably automatically generated by thelearning process without the need of a computer to do so. CogniMemdeposited on a same substrate can use identical or different knowledge.

CogniMem can be deposited on or embedded in (or otherwise coupled to) asubstrate as part of a CogniSensor or stand-alone. In the former case,the CogniMem is typically dedicated to recognizing pixel datatransmitted by a photosensitive element. In the latter case, theCogniMem may be used to support other CogniMem's and may be used, forexample, to recognize different data types transmitted by other CogniMemunits (for example to consolidate a pattern of responses from multipleCogniSensors).

The following listed patents and published applications, the entirecontents of each of which are hereby incorporated by reference, describevarious aspects of neuron and neural networks applicable to CogniMemsand CogniSensors: U.S. Pat. No. 5,621,863—Neuron Circuit; U.S. Pat. No.5,717,832—Improved neuron circuit architecture; U.S. Pat. No.5,701,397—Circuit for pre-charging a free neuron circuit; U.S. Pat. No.5,710,869—Daisy-Chain circuit for serial connection of neuron circuits;U.S. Pat. No. 5,740,326—Circuit for searching/sorting data in neuralnetworks U.S. Pat. No. 6,332,137—Parallel associative memory for astand-alone hardware recognition; U.S. Pat. No. 6,606,614—Single wiresearch and sort; Japanese applications JP8-171543—Daisy-Chain circuitfor serial connection of neuron circuits; JP8-171542—Advanced loadingcircuit; JP8-171541—Aggregation Circuit (Search/Sort); JP8-171540—NeuralNetwork and Neural chip; JP8-069445—Neuron circuit architecture; Koreanpatent application KR164943—Innovative neuron circuit architecture;European patents EP0694852—Innovative neuron circuit architecture;EP0694854—Improved neural semiconductor chip architecture;EP0694855—Search/Sort for neural networks; EP0694853—Circuit forpre-charging the input vector components in a free neuron circuit duringthe recognition phase; EP0694856—Daisy-Chain circuit for serialconnection of neuron circuits; Canadian application CA2149478—Improvedneuron circuit architecture; Canadian patent CA2149479—Improved neuralsemiconductor chip architecture

The number of neurons implemented on a CogniMem can vary from 1 to N,with N theoretically unlimited due to the architecture of the neuroncell. Currently, N can be as high as about 1000. In general, N isdetermined by the application and in particular, from the diversity ofpatterns to be recognized and the type of decisions to transmit. Oneskilled in the art will recognize that the silicon technology may be asignificant factor determining the number of neurons that can beprovided per unit area.

An exemplary configuration of an image recognition device according toan embodiment of the present invention is illustrated in FIGS. 1A and1B. FIG. 1A is a top view of the device 100, which includes a substrate102 that can be made from a number of transparent or semi-transparentmaterials such as glass, plexiglass, transparent plastics, etc. One ormore CogniSensors 104 (in this case, as an array) may be embedded intothe substrate 102 or, as in this case, attached or glued to or otherwisecoupled to a surface of the substrate 102 (See FIG. 1B). An optical pathcan be etched or deposited in front of each photosensitive element onthe substrate. For example, the substrate 102 can be etched at thelocation of the CogniSensors 104 in order to create lenses 102 a foreach CogniSensor 104. Alternatively, a microlens 102 a can be insertedinto the substrate 102 (FIG. 2) or glued (FIGS. 3 A-B) onto thesubstrate 102 in front of the photosensitive elements. Another optionmay be to alter the substrate to vary the reflective index of theportion of the substrate proximate each sensor, to focus incident light.As shown in FIG. 1B, incident light is focused on each CogniSensor 104by the substrate lenses 102 a.

The plurality of lenses 102 a allows the CogniSensors 104 to cover avariety of fields of view, preferably equal to the substrate surface butmay also possibly cover views narrower or larger than the field of viewequal to the substrate surface. The microlenses 102 a turn the array ofCogniSensors 104 into a telecentric image perception device with anunlimited surface and view.

FIG. 2 is a top view of a monolithic imaging device according to anotherembodiment of the present invention. As shown, lenses 102 a are embeddedinto substrate 102 and positioned over each CogniSensor 104. As anexample of a use of the imaging device, DNA fragments 202 are shownbeing positioned on the surface of the substrate 102. Each CogniSensor104 could be configured to recognize individually, or in collaborationwith adjacent CogniSensors 104, a particular DNA fragment and output asignal when that fragment is identified.

FIGS. 3 A-B illustrate an exemplary embodiment of an individualCogniSensor 104. As shown in FIG. 3A, an area of concentrated neurons104 a surrounds a pixel sensing region 104 b. The neurons in neuron area104 a can be coupled to sensing elements in pixel area 104 b and can beconfigured to recognize patterns sensed by the pixel area 104 b. Asshown in FIG. 3B, a convex lens or micro-lens 102 a is positioned overthe pixel area 104 b on the surface of a substrate 102 for focusingincident light onto the pixel area 104 b or connected directly to thesensor without an intermediate substrate. Lens 102 a could, for example,be chemically deposited onto the substrate by conventional means.

FIG. 4 is a functional block diagram of an exemplary CogniSensor 104according to an embodiment of the present invention. CogniSensor 104includes a sensor or sensing region 402, data presentation logic 404, aneural network 406, and local decision logic 408. The sensor 402 mayinclude one or more sensing elements, such as photosensitive elements.The data presentation logic 404 is coupled to the sensing region 402 andthe neural network 406 and is configured to present the data output fromthe sensors to the neurons in a manner suitable for processing. Theneurons 406 are or become “taught” with knowledge and can process datainput to neurons 406 from the presentation logic 404, and outputprocessed data to the local decision logic 408, which makes a decisionbased on the processed data. Local decision logic 408 may be coupled toother CogniSensors or CogniMem by various known methods. Accordingly,CogniSensors 104 may be arranged in arrays and arrays of arrays.

FIGS. 5A and 5B show arrangements of arrays of CogniSensors. As shown inFIG. 5A, each CogniSensor 104 can be coupled to a plurality ofCogniSensors 104 to for an array 502. As described below, input andoutput buses may be utilized for coupling of sensors in series orparallel.

As shown in FIG. 5B, each array 502 may be coupled to a plurality ofarrays 502 to form a bank of arrays 504. By arranging arrays of arraysof CogniSensors 104, an extremely powerful recognition device isproduced, that is both high-resolution and high-speed. That is, theresolution of the imaging device can be increased by increasing thenumber of sensors. However, by providing robust local decisioncapability in the form of CogniMem, the increase in the number ofCogniSensors does not decrease processing speed of the device. Further,one will understand that the arrays can be organized in many differentgeometries and the invention is not limited to square arrays.

As mentioned above, each neuron can be coupled to a plurality of inputs1-n, which can be, for example, multiplexed inputs, but is not limitedthereto. FIG. 6A is a representation of a neuron having multiple inputs,which is simplified in FIG. 6B. As a result, an array of neurons can beassembled using an input bus 602 (there is no bus 602 on FIG. 6C), asshown in the simple parallel architecture in FIG. 6C. Each output of theneurons 406 can be connected to a global decision bus 406.

FIG. 7 is a functional block diagram of an exemplary neuron according toan embodiment of the present invention. The purpose of the neuronsorganized as an unlimited expansion network is to learn and recalldigital vectors or signature (the pattern). Digital signatures aremostly spatial distributions of light intensity coded by a datareduction process. Neurons may be connected in parallel as representedin FIG. 6C, which means that all the neuron inputs are connected inparallel as well as all their outputs.

Data signals may be input from a multiplexed input bus (not shown) intothe neuron 700. A learn multiplexer 702 can divide the multiplexed inputsignals and transmit input data signals into a neuron recall memory 704and an associative logic element 706. The neuron recall memory 704processes the input signals and outputs processed signals to theassociative logic element 706. The associative logic element 706includes a similarity factor deciding element 706 a.

Every neuron can receive a broadcasted pattern (i.e., vectorrepresenting a digital signature of the sensor data) generated by thedata presentation logic 404. This broadcasted pattern can be a transform(data reduction) of sensor generated data either instantaneous, or inthe time domain.

A neuron has three possible subsequent chronological states: dormant,ready to learn (RTL) and thereafter committed. At least one neuron is inthe RTL state at all times except if the network is full (i.e., all theneurons being committed). If one considers all the parallel connectedneurons as a chain, the RTL neuron can move from the first position ofthe chain to the last position. In context of this representation, theRTL neuron will be typically on the right side of the committed neuronand the dormant neuron will be on the right side of the RTL neuron.

When a neuron is dormant, it will not react to any incoming pattern. ARTL neuron will load the incoming pattern into its recall memory inorder to learn it if the user process decides so. This RTL neuron willhave no participation in the recognition process but will be dedicatedto build new knowledge while learning.

The learning process includes creating new knowledge when an unknownpattern occurs and the user decides to learn it. This knowledge additionwill take place in the RTL neuron. In addition to creating a newknowledge, the committed neurons, which possibly wrongly identify anincoming pattern (i.e., fails to associate the proper category) willreduce their similarity domain to avoid further misclassification. Thiscauses knowledge modification or “adaptive learning.”

Photo elements can output a digitized radiometric value. The combinationof all the values across a spatial distribution forms a pattern; suchpattern can also evolve in the time domain and generate a stream ofpattern. This pattern goes through a data reduction process which leadsto the digital signature (vector) of the radiometric pattern. Thereduction process must not exceed what is called the “minimumdiscrimination matrix” described below. For example with a 5×7 matrix,it is possible to discriminate all the European uppercase characters butnot a Chinese Kanji character, for which a 16×16 matrix is needed.

A committed neuron learns a pattern when it is in the RTL state, byassociating the vector loaded into the recall memory 704 with a categoryheld into the category register 709. When the incoming pattern enters acommitted neuron the learn/reco mux 702 will let transmit it to theassociative logic 706 in order for this pattern to have its similarityevaluated toward the vector held into the recall memory 704. If thecalculated similarity is found to be less or equal the similarity factor706 a, the neuron will be excited and therefore signal thru the logic712. The function of the excitatory/inhibitory logic is to perform aglobal arbitration as many neurons become excited, among all thecommitted “firing” (i.e., excited) neurons and to “inhibit” thoseneurons which do no have the best similarity.

Region of Interest

Each CogniSensor may be associated with a region of interest (ROI) for avideo frame. Each CogniSensor can extract a signature of the ROI tobroadcast to its neurons (for learning or recognition purposes). Thesignature of the ROI is a compressed format of its pixel values reducedto fit into a sequence of N values with N being the size of the neuron'smemory cells.

Take the example where a neuron is outfitted with a memory capacity of256-bytes. A CogniSensor may classify a rectangular ROI of N×M pixels.The ROI signature will be reduced from N×M values to 256 values by, forexample, simple block compression.

CogniSensors can be configured to process ROIs of any shape, and achoice of signature extractions can be application specific (e.g., partinspection, surface inspection, face recognition, target tracking, etc).Some signature extractions can integrate time, repetitivity, etc. Also,neurons can be outfitted with memory cells larger than 8-bit toaccommodate inputs from sensors with 12-bit pixel resolution or more.

The combination of the neurons together with the sensor and the datapresentation logic constitutes a totally novel approach for embeddedimage recognition without any software needed for either the learning orthe recognition process.

The addressing of the CogniMem can be pass-through or selective (such asdriven by the response of other CogniMem units).

It should be understood that a substrate hosting CogniSensor(s) servesas both a mechanical support and as a lens (See, e.g., FIGS. 1-2). Thesubstrate can be, but is not limited to, a rigid or flexible, flat orcurved, surface made of a glass, Plexiglas, plastic, Mylar or othermaterial.

The connectivity between CogniSensors and CogniMem units on a samesubstrate should preferably use a minimum number of wires.

The knowledge loaded in the CogniSensors can preferably address therecognition of different families of patterns, whether related or not.

Examples

According to an embodiment of the present invention, CogniSensors areideal for performing inspection during an automated manufacturingprocess. As shown in FIG. 8, one or more CogniSensors could be used toinspect a water bottle. In this example, three different CogniSensorsare used to inspect three different regions referenced as Expert 1-3.The global response can depend on the combined responses of the three“expert” CogniSensors.

In this example, CogniSensor 1 (Expert 1) can be trained to classifysignatures of the ROI containing the cap of the bottle 802. CogniSensor1 can classify its ROI into 2 categories: Good and Bad. The Bad categorycan combine several cases: the cap is missing or the cap is not screwedon properly.

Similarly, CogniSensor 2 (Expert 2) can learn signatures of the ROIcrossing the line of fluid in the bottle 804. The ROI can be a narrowvertical rectangle and would ideally cover the minimum and maximumpossible filling levels in the bottle. Depending on the quality controlcriteria of the manufacturer, CogniSensor 2 can classify its ROI intoany number of categories, for example: Acceptable and Not Acceptable;Too High, Acceptable and Too Low; or Too High, High but Acceptable, InRange, Low but Acceptable, Too Low.

CogniSensor 3 (Expert 3) can learn signatures of the region of interestcovering the label area 806. CogniSensor 3 can be trained to recognize adiversity of cases or combination of cases such as for example: Missinglabel, Defective label (torn, scratched or folded), misplaced labels (upside down, slanted) and Good.

An output from CogniSensors 1-3 could be provided to controllerassociated with the automated manufacturing process to take appropriateaction based on the decisions made thereby.

According to an embodiment of the present invention, CogniSensors can beindividually packaged to form a smart photocell or smart microlens. Sucha device has application to a large number of technologies and could beused, for example, to detect moving parts, identify routes or routemoving parts in a mechanized assembly process (FIG. 9A); for biometricidentification, such as in a camera phone (FIG. 9B); or for visitordetection and identification in a door peep hole or the like (FIG. 9C).

According to another embodiment of the present invention, a driverawareness detection system is provided. Referring to FIG. 10, one ormore CogniSensors 104 may be embedded in a windshield, dashboard flatpanel display, or headlight of a motor vehicle. CogniSensors 104 can betaught to recognize patterns that indicate when a driver is no longerattentive (e.g., the driver is falling asleep) and output a signal totrigger an alarm. Such patterns could include gaze tracking, facerecognition, facial expression recognition and more. Further,CogniSensors 104 in a windshield or headlight could be taught torecognize objects or events external to the vehicle, such as foridentifying rain drops with a windshield wiper system or road hazardsfor a road hazard warning system.

The detection of an object which can appear randomly in the far or nearfield of view can be made a number of ways. For example, two or threesensors can be outfitted with lenses focusing at different distances.The sensors can be loaded with the same knowledge, but work on regionsof interest with different sizes. The global response of the recognitionsystem can be considered positive if at least one sensor recognizes theobject.

Also, CogniSensors can be designed with input sensors sensitive todifferent wavelengths such as Near-IR, IR, color filtered, etc. For agiven object or scene, such CogniSensors will produce different pixelvalues but can be trained on their respective video image to recognizethe categories of objects. In target tracking, the combination ofnear-IR and IR CogniSensors will give the ability to recognize a targetat any time of the day.

According to another embodiment of the present invention, arrays ofCogniSensors can be used in many other manufacturing applications. Forexample, as shown in FIG. 11A, a 1-dimensional array of CogniSensors1102 can be used to for inspection of glass floats 1103 in amanufacturing process. As shown in FIG. 11B, a 2-dimensional array ofCogniSensors 1104 can be used for detection of contaminants at thebottom of containers 1105, such as beverage bottles. In suchapplications, each CogniSensor can be taught to identify patterns thatindicate flaws in glass or contaminants in a fluid.

According to another embodiment of the present invention, CogniSensorscan be distributed across a glass plane or the like, to perform multipleindependent functions. CogniSensors can be grouped and taught withdifferent knowledge per group. FIG. 12 shows as one example, a slidingglass door 1202 that includes several groups of CogniSensors 1204 fordetecting approaching objects of different size. A first group could betaught with knowledge for recognizing a first size 1208 of human oranimal (e.g., dog), while a second group can be taught for a differentsize person (e.g., boy) 1210, a third group for another size person(e.g., adult) 1212, and so forth. Each group 1204 could be coupled toone or more CogniMems 1206 for control of the sliding door.

According to another embodiment of the present invention, as shown inFIGS. 13A-13D, one or more light elements 1301 may be embedded in orpositioned on the same substrate 102 in or on which one or moreCogniSensors 104 are embedded or positioned. Each CogniSensor 104 has afield of view from which light incident on the Cognisensor 104 isreceived. A light element 1301 may be configured to emit light at leastinto the field of view of one or more CogniSensors 104 (e.g., configuredto emit light through substrate 102). Alternatively, a light element1301 may be configured to emit light at least outside a field of view ofone or more CogniSensors 104 (e.g., configured to emit light away fromsubstrate 102). In some embodiments, one or more light elements 1301 maybe configured to emit light to one side of substrate 102, and one ormore other light elements 1301 may be configured to emit light to theother side of substrate 102.

The one or more light elements 1301 may be, without limitation, lightemitting diodes (LEDs), organic LEDs (OLEDs), plasma cavities or anyother suitable electronic or chemical luminescent source. The one ormore light elements 1301 may be driven to emit light having apredetermined intensity and/or color. The intensity and/or color of oneor more light elements 1301 may be different than the intensity and/orcolor of light emitted by one or more other light elements 1301. Inaddition, one or more light elements 1301 may be configured so that theintensity and/or color of emitted light are adjustable or controllable.As few as one CogniSensor 104 and one light element 1301 may be embeddedin or positioned on substrate 102. However, a plurality of CogniSensors104 and/or a plurality of light elements 1301 may be embedded in orpositioned on substrate 102. When a plurality of CogniSensors 104 and/orlight elements 1301 are used, light elements 1301 and Cognisensors 104may form an array on substrate 102, and the number of light elements1301 may or may not correspond to the number of Cognisensors 104. Forexample, there might be a certain number of light elements 1301 forevery one Cognisensor 104. The one or more light elements 1301 may beconfigured to provide lighting and/or display images.

As described above and shown in FIGS. 3A and 3B, each of the one or moreCogniSensors 104 may include a pixel area 104 b and a neuron area 104 a.Pixel area 104 b may receive light incident on pixel area 104 b from afield of view of the CogniSensor 104, and neurons in the neuron area 104a may be configured to detect and recognize patterns of the lightreceived by pixel area 104 b. Each of the one or more light elements1301 may be connected to and controlled in accordance with the output(s)of one or more CogniSensors 104. Accordingly, each of the one or morelight elements 1301 may be controlled in accordance with the patternsrecognized by one or more CogniSensors 104. For example, one or moreCognisensors 104 may control light elements 1301 to provide intelligentlighting and/or intelligent display.

An array of CogniSensors 104 and light elements 1301 that may beembedded in or positioned on substrate 102 is shown in FIG. 13A. In thisembodiment, CogniSensors 104 and light elements 1301 are arrayed in rowsand columns of alternating CogniSensors 104 and light elements 1301.Further, because CogniSensors 104 and light elements 1301 alternate inboth rows and columns, one or more of Congisensors 104 are adjacent to alight element 1301 on each of the four sides of the CogniSensor 104.Similarly, one or more of the light elements 1301 are adjacent to aCogniSensor 104 on each of the four sides of the light element 1301. Inalternative embodiments, CogniSensors 104 and light elements 1301 may bearrayed in alternating columns or rows of CogniSensors 104 and lightelements 1301. In these embodiments, there may be a one to one ratio oflight elements 1301 to CogniSensors 104. Further, each CogniSensor 104may control the intensity and/or color of an adjacent light element 1301and/or non-adjacent light elements 1301.

An array of CogniSensors 104 and light elements 1301 that may beembedded in or positioned on substrate 102 according to anotherembodiment is shown in FIG. 13B. In this embodiment, one or moreCogniSensors 104 may be embedded in or positioned on a substrate 102,and each CogniSensor 104 is surrounded by a plurality of light elements1301. For example, and as shown in FIG. 13B, each CogniSensor 104 may besurrounded by twelve light elements 1301. However, other ratios arepossible. For instance, each CogniSensor 104 may be surrounded by eightlight elements 1301. Although light elements 1301 may form a single ringaround each CogniSensor 104, one or more additional rings of lightelements 1301 may also be formed around each CogniSensor 104. EachCogniSensor 104 may control the intensity and/or color of the lightelements 1301 that surround the CogniSensor 104 and/or light elementsthat surround other CogniSensors 104.

FIG. 13C shows a cross-section of CogniSensors 104 and light elements1301 embedded in or positioned on a transparent or semi-transparentsubstrate 102. In this embodiment, light elements 1301 emit light to theside of substrate 102 from which light incident on CogniSensors 104 isreceived. Accordingly, light elements 1301 may be configured to emitlight through substrate 102 into the field of view of one or moreCogniSensors 104. As illustrated, lenses 102 a are embedded in orpositioned on substrate 102, and each lens 102 a provides an opticalpath to either a CogniSensor 104 or a light element 1301. Lenses 102 amay be formed in any of the manners discussed above in regard to FIGS.1B, 2 and 3B. Further, as shown in FIG. 13C, CogniSensors 104 and lightelements 1301 may alternate and the filler of filler layer 1310 may belocated between the alternating CogniSensors 104 and light elements1301. Filler layer 1310 may include conductors directly engraved ordiffused on substrate 102 configured to supply power and/or carrysignals to and/or from Cognisensors 104 and to and/or from lightelements 1301. The one or more CogniSensors 104 and/or one or more lightelements 1301 may be interconnected together by a transparent or opaquewiring. Also, a cover layer 1311 may be located on the side ofCogniSensors 104 and light elements 1301 opposite the substrate 102.Like substrate 102, cover layer 1311 may be transparent orsemi-transparent and may be made from glass, plexiglass or transparentplastics.

FIG. 13D shows a cross-section of CogniSensors 104 and light elements1301 embedded in or positioned on transparent or semi-transparentsubstrate 102 according to an embodiment in which light elements 1301emit light to a side of substrate 102 opposite the side of substrate 102from which light incident on CogniSensors 104 is received. Accordingly,light elements 1301 may, for example, emit light through cover layer1311 instead of through substrate 102. As shown in FIG. 13D, lenses 102a may be embedded in or positioned on substrate 102 and cover layer1311. Each lens 102 a embedded in or positioned on substrate 102provides an optical path to a CogniSensor 104, and each lens 102 aembedded in or positioned on cover layer 1311 provides an optical pathto a light element 1301. A lens 102 a may be embedded in or positionedon cover layer 1311 in any of the manners discussed above in whichlenses 102 a are embedded in or positioned on substrate 102.

All of the light elements 1301 shown in FIG. 13C are configured to emitlight through substrate 102, and all of the light elements 1301 shown inFIG. 13D are configured to emit light through cover layer 1311. However,in some embodiments, one or more light elements 1301 may be configuredto emit light through substrate 102 while other light elements areconfigured to emit light through cover layer 1311.

In operation, the outputs of one or more CogniSensors 104 may be used tocontrol one or more light elements 1301. As described above, the one ormore CogniSensors 104 may be used to detect and recognize patterns oflight incident on pixel areas 104 b of CogniSensors 104. Thus, one ormore CogniSensors 104 may be trained to detect the presence of one ormore objects within the CogniSensors' field of view, to count the numberof objects with the field of view, to determine the distance between oneor more detected objects and substrate 102 or CogniSensor 104 (e.g., viastereoscopy), to locate the position of one or more detected objects, todetermine whether one or more of the detected objects are authorizedobjects, to recognize faces within the field of view, to recognizefacial expressions on the recognized faces, to locate and track the gazeof one or more viewers within the field of view, and/or to performbiometric identification. In addition, one or more CogniSensors 104 maycontrol one or more light elements 1301 when detecting an unknownobject. Although not listed exhaustively herein, other forms of patternrecognition known to those skilled in the art are also possible.

The patterns recognized by one or more CogniSensors 104 may then be usedto selectively and individually control each of the one or more lightelements 1301. Examples of selective controlling of one or more lightelements 1301 may include turning one or more light elements 1301 on oroff, adjusting the intensity of light emitted by one or more lightelements 1301, and/or adjusting or changing the color of light emittedby one or more light elements 1301.

In addition, because the CogniSensors 104 may be spread out on or in thesubstrate 102, the CogniSensors 104 may have a large sensing areawithout the necessity of a wide angle lens. Accordingly, CogniSensors104 may detect and recognize objects even when the objects are veryclose to the substrate 102.

FIGS. 14A-14C illustrate an example configuration of light elements 1301and CogniSensors 104 in an intelligent lighting unit 1400. Like theembodiments shown in FIGS. 1-7, intelligent lighting unit 1400 has atransparent or semi-transparent substrate 102, and one or moreCogniSensors 104 are embedded in or positioned on substrate 102. Inaddition, one or more light elements 1301 may be embedded in orpositioned on substrate 102. In some embodiments, a cover layer 1311 maybe provided behind the one or more CogniSensors 104 and one or morelight elements 1301, and a filler layer 1310 may be provided betweensubstrate 102 and cover layer 1311. By having one or more CogniSensors104 and one or more light elements 1301 in or on the same substrate 102,the intelligent lighting unit may be thin and compact.

As shown in FIG. 14A, CogniSensors 104 and light elements 1301 may forman array on or in substrate 102. Although a specific example in whichgroups of CogniSensors 104 and light elements 1301 are arrayed in or onsubstrate 102 is shown in FIG. 14A, many other arrangements arepossible. For example, CogniSensors 104 and light elements 1301 may beuniformly distributed over substrate 102, may be clustered together, maybe located peripherally, and/or may be located centrally.

FIG. 14B shows a cross-section of intelligent lighting unit 1400, andFIG. 14C shows an exploded image of a portion of the cross-section shownin FIG. 14B having a CogniSensor 104 and light elements 1301. Although aspecific example configuration is shown in which light elements 1301 arelocated on either side of CogniSensor 104, other configurations arepossible. For instance, CogniSensor 104 may be surrounded on all sidesby light elements 1301 forming a square, rectangle, circle and/or ovalaround CogniSensor 104, and/or the Cognisensors 104 and light elements1301 may alternate. Alternatively, there may be only one CogniSensorand/or one light element 1301. Further, CogniSensor 104 may include anarray of CogniSensors as shown in FIG. 5A or may include an array ofCogniSensor Arrays as shown in FIG. 5C. As shown in FIG. 14B,intelligent lighting unit 1400 may also include a filler layer 1310located between substrate 102 and cover layer 1311. Filler layer 1310may include conductors directly engraved or diffused on substrate 102configured to supply power and/or carry signals to and/or from one ormore Cognisensors 104 and to and/or from one or more light elements1301.

As shown in FIG. 14C, CogniSensors 104 may receive light transmittedthrough transparent or semi-transparent substrate 102, and lightelements 1301 may emit light through substrate 102. However, one or morelight elements 1301 may be emit light though cover glass 1311 instead.

In operation, an intelligent lighting unit 1400 may control the lightemitted from light elements based on patterns detected in the field ofview of CogniSensors 104. For example, intelligent lighting unit 1400may turn on or increase the intensity of light emitted by one or morelight elements 1301 in response to CogniSensors 104 detecting that oneor more people are present in the CogniSensors' field of view.Similarly, intelligent lighting unit 1400 may turn off or dim (i.e.,reduce the intensity of light emitted by) one or more light elements1301 in response to CogniSensors 104 detecting that no people arepresent in the CogniSensors' field of view.

One or more light elements may be controlled in response to behaviorrecognized by intelligent lighting unit 1400. For example, anintelligent lighting unit 1400 may recognize that a person has opened abook and, in response, increase the intensity of light directed towardthe book. An intelligent lighting unit may turn off or dim one or morelight elements 1301 in response to one or more CogniSensors 104recognizing that one or more persons in the field of view have laid downin a bed. If one or more CogniSensors 104 in an intelligent light unit1400 has been trained to recognize when one or more persons havecollapsed, the intelligent light unit could change the color of lightemitted by the one or more light elements 1301 (e.g., change the colorof light emitted to red).

Further, the results of the pattern recognition may be transmittedoutside of intelligent lighting unit 1400. An intelligent lighting unit1400 that has recognized that a person has collapsed may notifyemergency personnel. In a hotel setting, in addition to dimming orturning off one or more light elements 1301, intelligent lighting unit1400 could inform the front desk and/or cleaning staff that people areno longer in the room. Also, intelligent lighting unit 1400 could signalfans, air conditioners and/or heaters to shut off when guests are nolonger present.

In a prison setting, intelligent lighting unit 1400 may be used torecognize and output the number of inmates in a cell. Intelligentlighting unit 1400 may be used to recognize when a fight is in progressor when an inmate needs medical attention. Intelligent lighting unit1400 could also set off an alarm, change the color of the light emittedby one or more light elements 1301, and/or notify prison staff when thenumber of inmates in a cell is more or less than an expected amount, afight is in progress, or medical attention is needed.

Furthermore, because intelligent lighting unit 1400 performs patternrecognition locally, transmission of images or video is not necessary,and intelligent lighting unit 1400 recognizes behavior without intrudingon people's privacy.

Intelligent lighting unit 1400 may also be used in a window pane in agreenhouse or other building. One or more CogniSensors 104 may detectthe amount of light outside the building and control one or more lightelements 1301. For instance, an intelligent light unit 1400 may be usedin a greenhouse to turn on one or more light elements 1301 to provideadditional light on cloudy days while keeping the one or more lightelements 1301 turned off on sunny days. Also, intelligent light unit1400 may be used as a window on the ceiling or wall of a building. Forinstance, when used as a window, the intelligent light unit 1400 maysimply appear as a transparent or semi-transparent window while the oneor more CogniSensors 104 receive more than a certain amount of light,but the intelligent light unit 1400 may provide lighting while the oneor more CogniSensors 104 receive less than the certain amount of light.In these embodiments, intelligent light unit 1400 may use theconfiguration of light elements 1301 and CogniSensors 104 shown in FIG.13D.

FIGS. 15A and 15B illustrate example configurations of one or more lightelements 1301 and one or more CogniSensors 104 in an intelligent display1500. Like the embodiments shown in FIGS. 1-7, intelligent lightingdisplay 1500 has a transparent or semi-transparent substrate 102, andone or more CogniSensors 104 are embedded in or positioned on substrate102. In addition, one or more light elements 1301 are embedded in orpositioned on substrate 102. Light elements 1301 are geometricallyarranged and configured to form images. The images formed may bedisplayed to objects in the field of view of one or more CogniSensors104. The images displayed may include a sequence of images (e.g.,video). Intelligent display 1500 may have an input/output unit throughwhich images to be displayed may be received and through which patternrecognition information may be sent. Input/output unit may send andreceive signals wirelessly. Here again, because the one or moreCognisensors 104 and light elements 1301 are in or on the same substrate102, the intelligent display 1500 may be thin and compact. Also, becauseCogniSensors 104 may be spread over the entire display, CogniSensors 104may have a large sensing area and be able to detect and recognizepatterns even when objects are standing very close to the substrate 102of the display 1500.

An array of CogniSensors 104 and light elements 1301 that may beembedded in or positioned on substrate 102 according to one embodimentof an intelligent display 1500 is shown in FIG. 15A. In this embodiment,CogniSensors 104 and light elements 1301 are arrayed in alternating rowsof CogniSensors 104 and light elements 1301. In this embodiment, pixels1301 a-1301 c are used as light elements 1301, and each pixel mayproduce light of a different color. For example, light elements 1301 mayinclude red pixels 1301 a, green pixels 1301 b and blue pixels 1301 c.Red pixels 1301 a may produce red light, green pixels 1301 b may producegreen light, and blue pixels 1301 c may produce blue light. EachCogniSensor 104 may control one or more sets of one or more red pixels1301 a, one or more green pixels 1301 b and one or more blue pixels 1301c so that, when added together, a broad range of colors may be produced.Each CogniSensor 104 may control the intensity and/or color of anadjacent set of pixels 1301 a-1301 c and/or non-adjacent sets of pixels1301 a-1301 c.

An array of CogniSensors 104 and light elements 1301 that may beembedded in or positioned on substrate 102 according to anotherembodiment of an intelligent display 1500 is shown in FIG. 15B. In thisembodiment, red pixels 1301 a, green pixels 1301 b and blue pixels 1301c are used as light elements 1301. CogniSensors 104 are embedded in orpositioned on transparent or semi-transparent substrate 102. Lenses 102a embedded in or positioned on substrate 102 may provide an optical pathfor to CogniSensors 104. A filler of filler layer 1310 may be locatedbetween CogniSensors 104. Pixels 1301 a-1301 c may be located on theside of filler layer 1310 opposite substrate 102. Pixels 1301 a-1301 cmay be configured to emit light away from substrate 102. A transparentor semi-transparent cover layer may be cover pixels 1301 a-1301 c sothat pixels 1301 a-1301 c and CogniSensors 104 may be located betweenthe cover layer and substrate 102.

Although a specific example of an arrangement of light elements 1301 andCogniSensors 104 is shown in FIG. 15, many other arrangements arepossible as will be recognized by persons skilled in the art. Forexample, pixels 1301 a-1301 c may form a triangle around eachCogniSensor 104. Also, there could be a plurality of sets of pixels 1301a-1301 c for each CogniSensor 104.

In operation, an intelligent display 1500 may control the light emittedfrom light elements based on patterns detected in the field of view ofone or more CogniSensors 104. For example, intelligent display 1500 mayrecognize one or more characteristics (e.g., gender, height, weight,age) of one or more viewers in the field of view of one or moreCogniSensors 104, and the characteristics may be used to displaytargeted advertisements accordingly. In one embodiment, intelligentdisplay 1500 tracks the gaze of one or more viewers and/or recognizesfacial expressions, and, in response, images providing more informationabout products in which a viewer is interested may be displayed.Further, one or more Cognisensors 104 may recognize that a viewer isbored with or dislikes one or more images currently being displayed, theinformation can be used to display different images to catch theviewer's attention and/or present different advertisements in which theviewer might be more interested. The patterns recognized by one or moreCogniSensors 104 may be output and used to determine what images,advertisements and/or products generated the most favorable responses inviewers.

In the context of storefront advertising, intelligent display 1500 mayallow a window shopper to see into the store through intelligent display1500, track the gaze of the store front shopper, and display information(e.g., promotional materials, price, available sizes, etc) about themerchandise at which the window shopper looks. Inside stores, anintelligent display 1500 may be used both to display advertisements andto recognize an occurrence of shoplifting.

Intelligent display 1500 may also be used as a mirror to render theimage received by one or more CogniSensors 104. Thus, a viewer lookingat intelligent display 1500 would see themselves. Intelligent display1500 could then modify the image for entertainment and/or advertisingvalue. For example, a viewer could be shown wearing an advertiser's newline of clothing and/or could be shown in another location.

One or more Cognisensors 104 of intelligent display 1500 may also beused to recognize the location of and distance from display 1500 ofvarious body parts of a viewer. This information may be used forinteractive video games or to control the intelligent display. Forexample, one or more CogniSensors 104 of intelligent display 1500 mayrecognize certain body movements as instructions to turn off thedisplay, to change the channel or input being displayed, or to adjustthe volume. In addition, because CogniSensors 104 may be arranged overthe entire substrate 102 and have a large sensing area, CogniSensors 104may detect and recognize objects even when located very close to thesubstrate 102.

Here again, because intelligent display 1500 performs patternrecognition locally, transmission of images or video is not necessary,and intelligent displays 1500 may recognize behavior without beingintruding on people's privacy. Accordingly, intelligent displays 1500may also be used to determine television ratings (e.g., how many peopleare watching and/or whether the viewer(s) is enjoying a program).

Intelligent display 1500 may also be used for intelligent eyewear toenhance the image seen by a user wearing the eyewear. For example, eyeglasses, sunglasses or contact lenses may have a transparent orsemi-transparent substrate in or on which one or more CogniSensors 104are embedded or positioned. CogniSensors 104 may be configured to detectand recognize patters in the light received by the CogniSensors 104.Light elements 1301, which may be pixels 1301 a-1301 c, may becontrolled in accordance with the patterns recognized by theCogniSensors 104 to enhance the images seen by the wearer of theintelligent eye wear. For example, intelligent eye wear may be used tohighlight possible weapons or threats, display names of recognizedfaces, display distances to recognized objects, highlight moving objectsor roads, add symbols and/or display names of landmarks. As a result,the wearer of the intelligent eyewear may see an augmented reality, andthe eyewear may provide an infinite screen (i.e., patterns may bedetected and recognized regardless of the direction in which the wearerlooks).

In any of the embodiments described herein, substrate 102 may be a flatplane oriented horizontally or vertically but may also by curved and/ororiented at any angle. Power may be supplied wirelessly by means of oneor more photovoltaic devices embedded in or attached to substrate 102.Also, each Cognisensor 104 may communicate over the same line(s) used tosupply power to the Cognisensor 104. Further, output transmission linesand/or power supply lines may be directly engraved or diffused onsubstrate 102.

As should be apparent to one skilled in the art, the imaging device ofthe present invention may be useful in innumerable other applicationsnot listed here. For example, another application includes permanentdamage detection (texture change) in dam, bridge or other manmadeconstruction. Implementation of such application should be apparent fromthe above description of embodiments of the present invention. Further,power and signal transmission could be wireless (e.g., infra red,photocell, induction loop, etc.).

Thus, a number of preferred embodiments have been fully described hereinwith reference to the drawing figures. Although the invention has beendescribed based upon these preferred embodiments, it would be apparentto those of skill in the art that certain modifications, variations, andalternative constructions could be made to the described embodimentswithin the spirit and scope of the invention.

What is claimed is:
 1. An image recognition device comprising: a sensingelement embedded in or positioned on a transparent or semi-transparentsubstrate; a processing element coupled to said sensing element, saidprocessing element being embedded in or positioned on the substrate; anda light element embedded in or positioned on the substrate; wherein saidtransparent or semi-transparent substrate constitutes an opticalinterface between an incident image to be sensed and a sensing pixel ofsaid sensing element; and wherein the light element is configured toemit light toward the incident image or away from the incident image. 2.The image recognition device of claim 1, wherein the light elementcomprises one or more light emitting diodes (LEDs), organic LEDs (OLEDs)or plasma cavities.
 3. The image recognition device of claim 1, whereinthe light element is controlled selectively by an output of theprocessing element.
 4. The image recognition device of claim 1, whereinthe processing element is trainable and configured to recognize patternsbased on the sensed incident image.
 5. The image recognition device ofclaim 4, wherein the processing element is configured to control thelight emitted by the light element in accordance with the patternsrecognized by the processing element.
 6. The image recognition device ofclaim 1, wherein the sensing element has a field of view, and the lightelements are configured to emit light in the field of view.
 7. The imagerecognition device of claim 1, wherein the sensing element has a fieldof view, and the light elements are configured to emit light in adirection away from the field of view.
 8. The image recognition deviceof claim 1, wherein said processing element comprises a plurality ofneurons coupled on an input side thereof by a multiplexed input bus andon an output side thereof by an output bus, each said neuron beingtaught with a knowledge, said knowledge allowing the correspondingneuron to recognize a signal and perform a decision.
 9. The imagerecognition device recited in claim 1, wherein the processing element isconfigured to perform image recognition operations digitally without asoftware program through a plurality of parallel elements each havingself contained, autonomous behavior.
 10. The image recognition device ofclaim 1, further comprising photovoltaic devices embedded in orpositioned on said substrate.
 11. The image recognition device of claim1, further comprising output transmission lines and power supply linesthat are directly engraved or diffused on said substrate.
 12. The imagerecognition device of claim 1, wherein the processing element isconfigured to receive power from power supply lines and to outputcommunications using the power supply lines.
 13. The image recognitiondevice of claim 1, further comprising a transparent or semi-transparentcover layer, wherein the sensing element, the processing element and thelight element are arranged between the cover layer and the substrate.14. The image recognition device of claim 14, wherein the light elementis configured to emit light through the substrate.
 15. The imagerecognition device of claim 14, wherein light element is configured toemit light through the cover layer.
 16. An image recognition methodcomprising: providing an optical path to a plurality of sensing elementsembedded in or provided on a transparent or semi-transparent substrateby using a plurality of optical interfaces embedded in or provided onsaid substrate; processing, in parallel, signals generated from saidplurality of sensing elements in a plurality of processing elements eachcoupled to one of said sensing elements and each embedded in or providedon said substrate; and emitting light from a plurality of light elementsembedded in or provided on said substrate.
 17. The image recognitionmethod of claim 16, wherein the emitting comprises controlling the lightemitted from the plurality of light elements in accordance with outputsfrom one or more of the plurality of processing elements.
 18. The imagerecognition method of claim 16, wherein the processing comprisesrecognizing patterns and the emitting comprises controlling the lightemitted from the plurality of light elements in accordance with therecognized patterns.
 19. The image recognition method of claim 18,wherein the recognizing patterns comprises detecting the presence of oneor more objects within a field of view of said plurality of sensingelements.
 20. The image recognition method of claim 19, wherein therecognizing patterns comprises determining distance from said substrateof the one or more detected objects.
 21. The image recognition method ofclaim 19, wherein the recognizing patterns comprises determining thenumber of the one or more detected objects.
 22. The image recognitionmethod of claim 19, wherein the recognizing patterns comprises locatingthe position of the one or more detected objects.
 23. The imagerecognition method of claim 19, wherein the controlling comprisesemitting a reduced amount of light from a plurality of light elementswhen the presence of no objects is detected.
 24. The image recognitionmethod of claim 19, wherein the recognizing patterns comprisesdetermining whether any of the one or more detected objects is anauthorized object.
 25. The image recognition method of claim 18, whereinthe recognizing patterns comprises locating and tracking the gaze of oneor more viewers within a field of view of said plurality of sensingelements.
 26. The image recognition method of claim 18, wherein therecognizing patterns comprises facial recognition or facial expressionrecognition.
 27. The image recognition method of claim 18, wherein therecognizing patterns comprises biometric identification.
 28. The imagerecognition method of claim 16, wherein the emitting comprisesdisplaying an image.
 29. The image recognition method of claim 28,wherein the displayed image corresponds to an image received by theplurality of sensing elements.
 30. The image recognition method of claim28, wherein the processing comprises recognizing patterns and theemitting further comprises modifying the displayed image in accordancewith the recognized patterns.
 31. An image recognition methodcomprising: providing an optical path to a sensing element embedded inor positioned on a transparent or semi-transparent substrate by using anoptical interface embedded in or positioned on said substrate;processing signals generated from said sensing element in a processingelement coupled to said sensing element and embedded in or positioned onsaid substrate; and emitting light from a light element embedded in orpositioned on said substrate.
 32. The image recognition method of claim31, wherein the emitting comprises controlling the light emitted fromthe light element in accordance with an output from the processingelement.
 33. The image recognition method of claim 31, wherein theprocessing comprises recognizing patterns and the emitting comprisescontrolling the light emitted from the plurality of light elements inaccordance with the recognized patterns.
 34. The image recognitionmethod of claim 33, wherein the recognizing patterns comprises detectingthe presence of one or more objects within a field of view of saidsensing element.
 35. The image recognition method of claim 34, whereinthe recognizing patterns comprises determining distance from saidsubstrate of the one or more detected objects.
 36. The image recognitionmethod of claim 34, wherein the recognizing patterns comprisesdetermining the number of the one or more detected objects.
 37. Theimage recognition method of claim 34, wherein the recognizing patternscomprises locating the position of the one or more detected objects. 38.The image recognition method of claim 34, wherein the recognizingpatterns comprises determining whether any of the one or more detectedobjects is an authorized object.
 39. The image recognition method ofclaim 33, wherein the recognizing patterns comprises locating andtracking the gaze of one or more viewers within a field of view of thesensing element.
 40. The image recognition method of claim 33, whereinthe recognizing patterns comprises facial recognition or facialexpression recognition.
 41. The image recognition method of claim 33,wherein the recognizing patterns comprises biometric identification. 42.The image recognition method of claim 31, wherein the emitting comprisesdisplaying an image.
 43. The image recognition method of claim 42,wherein the displayed image corresponds to an image received by thesensing element.
 44. The image recognition method of claim 42, whereinthe processing comprises recognizing patterns and the emitting furthercomprises modifying the displayed image in accordance with therecognized patterns.